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Abstract —Loop closure detection, the task of identifying 
locations revisited by a robot in a sequence of odometry and 
perceptual observations, is typically formulated as a combi¬ 
nation of two subtasks: (1) bag-of-words image retrieval and 
(2) post-verification using RANSAC geometric verification. The 
main contribution of this study is the proposal of a novel 
post-verification framework that achieves good precision recall 
trade-off in loop closure detection. This study is motivated by 
the fact that not all loop closure hypotheses are equally plau¬ 
sible (e.g., owing to mutual consistency between loop closure 
constraints) and that if we have evidence that one hypothesis 
is more plausible than the others, then it should be verified 
more frequently. We demonstrate that the problem of loop 
closure detection can be viewed as an instance of a multi-model 
hypothesize-and-verify framework and build guided sampling 
strategies on the framework where loop closures proposed using 
image retrieval are verified in a planned order (rather than 
in a conventional uniform order) to operate in a constant 
time. Experimental results using a stereo SLAM system confirm 
that the proposed strategy, the use of loop closure constraints 
and robot trajectory hypotheses as a guide, achieves promising 
results despite the fact that there exists a significant number of 
false positive constraints and hypotheses. 

1. Introduction 

Loop closure detection, the task of identifying locations 
revisited by a robot in a sequence of odometry and per¬ 
ceptual observations, is an important first step to estimate 
robot trajectory and has contributed to important applications 
including localization & mapping [1], landmark discovery 
[2], image alignment [3], topological mapping [4], and 
place recognition [5]. Failure in loop closure detection can 
yield catastrophic damage in an estimated robot trajectory, 
and achieving an acceptable tradeoff between precision and 
recall is critical in this context. In previous research [5]- 

[7] , loop closure detection is typically formulated as a bag- 
of-words image retrieval problem where a query/database 
image is represented by an unordered collection of vector 
quantized local invariant features termed visual words and 
then efficiently indexed and retrieved to identify pairs of 
matched locations using each view image as a query input. 
However, even state-of-the-art image retrieval techniques 
generate a considerable number of false positives owing to 
confusing features and perceptual aliasing [5]. Therefore, 
the image retrieval process is typically followed by a post¬ 
verification step using robust RANSAC verification methods 

[8] . However, the post-verification step is a computationally 
intensive process, which requires quadratic time in the length 
of a view sequence. Therefore, improving the total cost 
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for performance of loop closure detection is an important 
practical problem and main focus of this study. 

In this study, we address the above issue with guided 
sampling. Unlike previous frameworks where latest loop 
closure constraints Zt (proposed by image retrieval) are 
verified individually in a uniform order at each location t, in 
the proposed framework, a constant number of constraints 
are intelligently sampled in a planned order from all the 
constraints obtained to date. This study is motivated 
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Eig. 1. Guided sampling for loop closure verification. Presented in (a) is 
our experimental environment with robot trajectory #1 (black), #2 (green), 
and #3 (orange). Loop closure detection is an essential task for correcting 
the accumulated error in visual odometry (b). Indicated in (c) are typical 
results of detecting loop closure constraints by FAB-MAP image retrieval 
-I- RANSAC post-verification, for different settings of RANSAC thresholds, 
TH=8,10,12,14,16. One can see that the detection performance in terms of 
precision and recall is significantly less than perfect and that high recall is 
achieved when TH= 8 or 10 at the cost of extremely low precision. The basic 
idea of guided sampling is to verify loop closure hypotheses in a planned 
order (rather than in conventional uniform order) by exploiting a domain 
specific knowledge of mutual consistency between loop closure constraints. 
Illustrated in (d) and (e) are sample results of guided sampling using two 
different strategies, called NN sampling and trajectory sampling. 







by the fact that not all loop closure hypotheses are equally 
plausible (e.g., owing to mutual consistency between loop 
closure constraints [9]) and that if we have evidence that one 
hypothesis is more plausible than the others, then it should 
be verified more frequently. Examples of such plausible 
hypotheses include: 

1) a loop closure constraint that provides a plausible 
reconstruction of a robot trajectory; 

2) a loop closure constraint that is consistent with a 
plausible robot trajectory hypothesis; 

3) a loop closure constraint that is spatially similar to a 
plausible loop closure constraint. 

To implement the above idea, we cast loop closure detec¬ 
tion as an instance of a multi-model hypothesize-and-verify 
problem where a set of hypotheses of robot trajectory is 
hypothesized from loop closure constraints and verified in 
terms of consistency against other loop closure constraints. 
The proposed approach is motivated by three independent 
observations. First, we are inspired by the recent success of 
guided sampling strategies in hypothesize-and-verify tech¬ 
niques (e.g., USAC [10]). Second, loop closure detection 
is essentially a multi-model estimation problem [11], rather 
than the single model estimation considered in classical 
applications of the hypothesize-and-verify approach (e.g., 
structure-from-motion [10]), where the goal is to identify 
multiple instances of models (i.e., loop closure hypotheses) 
and the inliers to one model behave as pseudo-outliers to the 
other models. Finally, and most importantly, the framework 
is sufficiently general and effective for implementing vari¬ 
ous guided sampling strategies that implement the domain 
knowledge presented above. 

Although the proposed approach is general, we focus on 
a challenging scenario of stereo SEAM, which has been 
attracting increasing interest in recent years [12], to demon¬ 
strate the efficacy of the proposed system. Our experiments 
employ a stereo SEAM system that implements odometry 
using stereo visual odometry as in [13], loop closure de¬ 
tection using appearance-based image retrieval with SURF 
local features and bag-of-words image model as in [5], post¬ 
verification using RANSAC geometric verification on local 
feature keypoints as in [8], and pose graph SLAM as in 
[14]. FigITJ),c illustrate an odometry-based robot trajectory, 
trajectory corrected by loop closing, and a set of loop closure 
constraints determined by employing the two strategies. As 
can be observed, significant errors in trajectory are accumu¬ 
lated as the robot navigates and the errors are successfully 
corrected given correct loop closure constraints. It can also 
be seen that image retrieval-based loop closure detection 
and RANSAC post-verification are both less than perfect 
(Fig. [It); there are a significant number of false positives 
and negatives. These two types of errors, accumulated er¬ 
rors in odometry and misrecognition in detection and post¬ 
verification of errors, are the main error sources that we 
address in this study. Experimental results using our stereo 
SLAM system confirm that the proposed strategy, the use 
of loop closure constraints and robot trajectory hypotheses 


as a guide, achieves promising results despite the fact that 
there are a significant number of false positive constraints 
and hypotheses (Fig. [J]). 

A. Related Work 

This study can be viewed as a novel application of multi- 
model-based consensus approaches (e.g., multi-RANSAC 
[11]). Similar to previous multi-model approaches, we focus 
on determining multiple plausible hypotheses (i.e., loop 
closure hypotheses) rather than the single best hypothesis. 

We allow a set of detected models (i.e., loop closure 
constraints) to be partially inconsistent with each other. 
Such partial inconsistency in loop closure constraints can 
be resolved reliably by employing modern SLAM back-ends 
such as robust pose graph optimization in [9]. 

Most of the existing works on loop closure detection 
have focused on the image retrieval step in the task, rather 
than the post-verification step [15]. In fact, loop closure 
detection techniques are typically classified in terms of image 
retrieval strategies (rather than post-verification strategies) 
[16]. Images are typically represented by a collection of 
invariant local descriptors [5] or a global holistic descriptor 
[7], [17]. Loop closure detection has been employed by many 
SLAM systems [18]-[21]. However, the above works did not 
focus on the post-verification step or introduce novel insight 
to the guided sampling strategy. 

Guided sampling has been studied in many matching 
problems that are closely related to loop closure detection 
with respect to its objective. In [22], techniques were pre¬ 
sented for improving the speed of robust motion estimation 
based on the guided sampling of image features, which 
is inspired by the MLESAC algorithm. [23] presented an 
approach called double-window optimization where a place 
recognition is used to identify loop-closing constraints and 
incorporate the constraints into the optimization. [24] ex¬ 
plored an approach based on considering features in the 
scene database and matching them to query image features 
as opposed to previous methods that match image features 
to visual words or database features, and presented an 
efficient solution based on prioritized feature matching. [25] 
proposed a method for improved geometric verification by 
exploiting the statistics of image collections and gathering 
information during geometric verification, to improve the 
overall efficiency. [26] presented an unsupervised learning 
approach for learning threshold for geometric verification. 
Recently, a series of robust SLAM back-end algorithms that 
allow misrecognition in loop closure constraints have been 
studied [9]. However, their objectives are neither the guided 
sampling nor SLAM front-end applications; rather, they fol¬ 
low conventional batch-style matching. Moreover, differing 
from general purpose matching algorithms, we are interested 
in and focus on the use of task specific knowledge regarding 
loop closure detection to improve overall performance. 

This paper is a part of our studies on loop closure 
detection. Recently, we have discussed cross-season place 
recognition [27], part-based scene modeling [28], landmark 
discovery [29], and map descriptor [30] in IROS15, ICRA15, 


PPNIV15 papers. Guided sampling in loop closure detection 
has not been addressed in the above papers. 

II. Approach 
A. Loop Closure Detection 

For clarity of presentation, we first describe a baseline 
SLAM system where the proposed approach is built and 
used as a benchmark for performance comparison in the 
experimental section. As mentioned, we build the proposed 
system on a stereo SLAM system where a stereo vision 
sensor is employed for both visual odometry [13] and visual 
feature acquisition [5] and follow the standard formulation 
of pose graph SLAM [31]. In pose graph SLAM, the robot 
is assumed to move in an unknown environment, along 
a trajectory described by a sequence of random variables 
XT. While moving, it acquires sequences of 
odometry measurements u\j= u\, •••, ut and perception 
measurements zv.t = z\, •••, zr- Each odometry measure¬ 
ment Ut (1 < ^ < r) is a pairing of rotation and translation 
acquired by visual odometry. Each perceptual measurement 
Zt is a set of loop closure constraints z], • • •, , each of 

which is a pair of location IDs, L t' with a likelihood score 
that represents the likelihood of the location pair belonging to 
the same place, which is acquired by FAB-MAP in our case. 
More formally, we begin with an empty list of loop closure 
constraints. At each time t, we execute FAB-MAP using the 
latest visual image as a query to identify the top = 50 
ranked images that receive the highest likelihood scores. We 
then insert the = 50 pairs from the query image and each 
of the Nt top-ranked database images as new constraints to 
the list. 

For simplicity, we begin by assuming that fixed sets 
of loop closure constraints zi-.t and map hypotheses m^M 
are a priori given; typical hypothesize-and-verify algorithms 
require such a fixed set assumption [11]. Clearly, this as¬ 
sumption is violated in our SLAM applications as both the 
loop closure constraints and the map hypotheses must be in¬ 
crementally derived as the robot navigates. This incremental 
setting will be addressed in Section UFO bv relaxing the fixed 
set assumption. We divide the entire measurement sequence 
into constant time windows and generate one hypothesis per 
window. To generate a hypothesis, we employ pose graph 
SLAM that expects the following as input: (1) the single 
loop closure constraint whose score received from FAB- 
MAP is the highest within the time window of interest, and 
(2) a sequence of previous odometry measurements. This 
yields M =T/W map hypotheses when the size of the time 
window is W. In experiments, we set the time window size 
sufficiently small, W = 10, that appearance of images do not 
change significantly within time window. 

Performance of loop closure detection is typically evalu¬ 
ated by precision-recall. This performance measure requires 
a set of ground truth loop closure constraints and a set of 
constraints verified as matched by the RANSAC. We run the 
RANSAC for each loop closure constraint (selected by the 
guided sampling) that consists of a pair of images to check 
if the keypoint configuration is geometrically consistent 


between the image pair. We use RANSAC verification with 
the fundamental matrix in [8] and a preset threshold of 
“8”, as indicated in Fig[T]:. If the RANSAC score exceeds 
the threshold, the input pair is verified as matched. For 
each query image /, we define a ran^e of ground-truth loop 
closure constraints in the form: •••, and 

consider a verified constraint (ij) is correct if and only if 

Based on the above terminology, we formulate the problem 
of guided sampling in loop closure detection. Let 5* denote a 
selection of a loop closure constraint. Recalling that a set of 
Nt new constraints arrive at each time instance t, we represent 
a selection 5' of loop closure constraints by: 

s={l,n) te[l,T],ne[l,Nt]. (1) 

Let Vi denote the result of post-verification: 

Vi = V{si) v,G{0,l}, (2) 

which indicates if the RANSAC score exceeds a predefined 
threshold (“1”) or not (“0”). Guided sampling is the problem 
of selecting the next constraint to verify: 

given a history of previous odometry ui-j and perception zi-.t, 
and a history of previous selections si-j-i, and verification 
results 

B. Guided Sampling Strategies 

A naive strategy for guided sampling is to uniformly sam¬ 
ple one constraint/hypothesis from a history of previously 
acquired constraints/hypotheses. This strategy is straightfor¬ 
ward and easy to implement. Unfortunately, it does not 
achieve acceptable precision-recall tradeoff as indicated in 
the experimental section (Section To achieve improved 
precision-recall tradeoff, we present several different strate¬ 
gies for guided sampling in the following discussion. 

The first strategy, termed trajectory sampling (TS), is a 
strategy that samples loop closure constraints that are con¬ 
sistent with a robot trajectory hypothesis (Fig. [it). The basic 
idea is to verify the consistency between a pair of constraints 
using the robot trajectory hypothesis as an intermediate. This 
strategy selects one random robot trajectory hypothesis h and 
then samples a loop closure constraint, a pair of IDs {t^') 
of locations that are near each other: 

\\p{t,h)-p{t',h)\\<Tp, (4) 



Fig. 2. All of the 538 robot trajectory hypotheses generated in Experiment 
#1. On the right is a close-up of the figure on the left. 











where p{t^h) is the 2-dimensional coordinate of location t 
conditioned on a robot trajectory hypothesis h, and Tp is a 
preset threshold, 10 m. 

The second strategy, termed neighbor sampling (NS), 
is a strategy that samples loop closure constraints that are 
neighbors to plausible constraints. Fig. [TJi presents examples 
of neighbor sampling. This strategy is motivated by the fact 
that in street-like environments, there often exists a sequence 
of matched views, rather than single isolated matches. This 
strategy selects one random trajectory hypothesis and one 
random verified loop closure constraint {ij) on the selected 
trajectory, and then samples one of its four neighbors (/ ± 
!)• 

The third strategy, termed breadth first (BF), is a strategy 
that samples trajectory hypotheses in a breadth-first order, 
rather than in a uniform order. This strategy is motivated 
by a limitation of uniform samplings in an incremental 
scenario, that is, the total number of old sampling hypotheses 
generated at the beginning of a robot’s navigation tends to 
be considerably greater than the new hypotheses generated 
at the end of the navigation. As a result, the distribution of 
“being sampled” becomes considerably unbalanced among 
the hypotheses as indicated in Fig. (db “UNIFORM SAM¬ 
PLING”. To address this limitation, this strategy continu¬ 
ously monitors the number of individual hypotheses being 
sampled to date and selects the hypothesis with the least 
number of being sampled as the next hypothesis to sample. 

The fourth strategy, termed depth first (DF), is a strat¬ 
egy that samples a trajectory hypothesis according to the 
importance weight, which is evaluated by the number of loop 
closure constraints that are consistent with the hypothesis of 
interest and have been verified as matched to date. Currently, 
we divide the hypothesis set into two subsets, upper half and 
lower half, according to the importance weight, and then 
sample the next hypothesis from the former subset. 

We need to select one guided sampling strategy at a 
time among the several different strategies for the sampling 
constraints (TS, NS) and hypotheses (BF, WS) discussed 
above. Currently, we introduce a random selection rule using 
preset parameters Pts, Pns^ Pbf and Pdf- We randomly 
select one of the three different strategies for hypothesis 
sampling, BF, DF, and uniform strategies with probabilities 
Pbf • Pdf • Pdf—Pbf)- We also randomly select one 
of the three different strategies for constraint sampling, TS, 
NS, and uniform strategies with probabilities Pts - Pns - 
(1—Pts—Pns)- In experiments, we test the different settings 
of these probabilities to investigate the contributions of the 
different sampling strategies. 

C. Incremental Extension 

In this section, we relax the fixed set assumption in Section 
III-AI and consider the general incremental setting of loop 
closure detection. As a main extension, the system must 
update the set of loop closure constraints and set of map 
hypotheses. Further, because sampling strategies introduced 
in Section HLAI relv on the knowledge of mutual consistency 
between loop closure constraints and map hypotheses, it also 


must update this knowledge every time a new constraint 
or hypothesis arrives. Thus, we introduce the concept of 
consistency matrix: C, which is an A/^T xM sized matrix 
defined as consistency Qj between each i-th loop closure 
constraint and j-th map hypothesis, and we incrementally 
update it every time a new constraint or hypothesis arrives. 
The update for a new constraint and hypothesis requires 
M and NtT computations of consistency, respectively. In 
the moderate-sized environments (e.g., M < 1000, NtT < 
1 X 10^) considered in the experimental section, we can 
assume these two costs are negligible compared to the costs 
for loop closure verification and map reconstruction. 

III. Experiments 

A. Settings 

We conducted loop closure detection experiments using a 
stereo SLAM system on a university campus. Our experi¬ 
ments employed a stereo SLAM system that implemented 
the proposed guided sampling strategies. The principal steps 
involved visual odometry, loop closure detection, and post¬ 
verification. The first step executed stereo visual odometry to 
reconstruct the robot trajectory. We adopted the stereo visual 
odometry algorithm proposed in [13], which has proven to 
be effective in recent visual odometry applications (e.g., 
[32]). The second step applied the appearance-based image 
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Fig. 3. Effectiveness of using robot trajectory hypotheses as a guide 
for sampling loop closure constraints, (a) is a histogram of the success 
ratio of detecting correct loop closure hypotheses for two distinct cases: 
sampling is guided (“PLANNED”) and not guided (“UNIFORM”). The 
horizontal axis represents the accuracy of the robot trajectory hypotheses, 
in terms of error [m] in robot location on the reconstructed trajectory with 
respect to the ground-truth trajectory, (b) compares success ratio for different 
combinations of guided sampling strategies. In the legend, “DF”, “BF”, 
and “US” are the depth -first, breadth-first, and uniform sampling strategies, 
respectively, in Section HLBI “x:y:z” indicates that the parameters Pts^ Pns 
for switching strategies are set to Pts = z/{x+y+z) and Pj^s — yl{^+y+z). 
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Fig. 4. Performance results. From left to right, the results for three different experiments, #1, #2, and #3, are presented. From top to bottom, each panel 
presents (a) the precision-recall curve, (b) a comparison of total number of individual hypotheses being sampled between uniform sampling and breadth 
first strategies, (c) ratio of loop closure constraints that are guided by individual strategies and also verified as matched by RANSAC (horizontal axis: time 
window ID), and (d) ratio of loop closure constraints that are guided by individual strategies, verified as matched by RANSAC, and are correct constraints 
with respect to the ground-truth (horizontal axis: time window ID). 


retrieval, FAB-MAP, proposed in [5]. This step generated a 
set of Nt new loop closure constraints and inserted these into 
the constraint list as indicated in Section III-AI The third step 
performed guided sampling to select a set of loop closure 
constraints to verify and applied each of these to RANSAC 
verification to be classified as matched or unmatched. 

Fig. [I]i,b present the ground-truth robot trajectories and 
visual odometry trajectories superimposed on Google map 
imagery. The ground-truth trajectories were generated using 
a SLAM algorithm based on the graph optimization in 
[14] using manually identified ground-truth loop closure 
constraints as input. As indicated, significant odometry errors 
were accumulated as the robot navigated. We collected three 
sequences along routes with travel distances of 1364 m, 1020 
m, and 1250 m, using a cart equipped with a Bumblebee 
stereo vision camera system, as illustrated in FigH^. We 


defined a ground-truth loop closure constraint as a pairing 
of two locations ij whose distance was less than 10 m. 
Occlusion was severe in the scenes and people and vehicles 
were dynamic entities occupying the scenes. We processed 
each path and collected stereo image sequences with lengths 
5759, 6358, and 6034. 

B. Proof-of-Concept Experiment 

In this paper, we rely on an assumption that robot trajec¬ 
tory hypotheses act as a guide to sample a new good loop 
closure constraint. One could argue that because an excessive 
number of false positive hypotheses exist, as indicated in Fig. 
[21 that the guide could cause more harm than assistance. As 
a proof-of-concept experiment, we experimentally compared 
the ratio of successful loop closure detection for three distinct 
cases: (1) not guided, (2) guided by a correct robot trajectory 
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Fig. 5. Examples of correct loop closure constraints proposed by two different guided sampling strategies. For each example, the top and bottom panel 
represent two different loop closure constraints = (ij) and = (/',/) using the corresponding image pairs {ij) and (/',/), where the former 

constraint is used as a guide to sample the latter (and correct) constraint 


hypothesis, and (3) guided by an incorrect robot trajectory 
hypothesis. To evaluate (2) and (3), we collected sets of robot 
trajectory hypotheses with different levels of correctness, 
which were measured in terms of percentage of correct 
loop closure constraints hypothesized by the hypothesis of 
interest. Fig. [3] presents the success ratio for loop closure 
detection guided by robot trajectory hypotheses. It can be 
seen that the proposed guided sampling does not harm the 
process, even when an inaccurate (false positive) hypothesis 
is used and that it does help when an accurate hypothesis 
is used. We can observe that sampling guided by false 
positive hypotheses frequently behaves similarly to uniform 
sampling, as the robot’s poses on false positive hypotheses 
typically distribute uniformly over the environment, as indi¬ 
cated in Fig. [21 

C. Precision-Recall Performance 

We evaluated the proposed strategies for guided sampling. 
We employed precision and recall as introduced in Section 
III-AI as the performance measure. We did not use trivial 
loop closure constraints where the travel distance between 
the image pairs was overly short, less than 100 m. That is, 
loop closures were not detected until the robot had travelled 
100 m from the start location in all experiments. 

FigSt presents the precision-recall curves. It can be ob¬ 
served that the TS strategy method outperformed all the other 
strategies considered in the current experiments. We observe 


that TS strategy was able to use robot trajectory hypotheses 
as an effective guide to propose good loop closure constraint, 
despite the fact that majority of robot trajectories are partially 
incorrect with respect to the ground truth as shown in Fig. 

El 

D. Examples of Guided Sampling 

Fig. [5] presents examples of correct loop closure con¬ 
straints proposed by guided sampling. We tested two differ¬ 
ent strategies: neighbor sampling, which uses a previously 
verified loop closure constraint as a guide and samples its 
neighbor constraints, and trajectory sampling, which uses 
robot trajectories reconstructed from previously verified loop 
closure constraints as a guide and samples constraints that 
are consistent with a reconstructed trajectory. It can be seen 
that the neighbor sampling strategy contributed to detecting 
loop closure constraints of similar scenes to previously 
detected loop closure constraints; whereas, the trajectory 
sampling strategy tended to detect loop closure constraints 
of extremely dissimilar scenes. 

E. Ratio of Verified/Matching Constraints 

Fig. [4|: and Fig. [4]l present the ratio of loop closure 
constraints that are verified as matched and are correct 
with respect to the ground-truth. It can be seen that the 
neighbor sampling strategy outperformed the other strategies 
considered in finding matched and correct constraints. We 


























observe that neighbor sampling strategy was effective to 
sample good constraints especially when almost all the robot 
trajectories reconstructed to date are incorrect with respect 
to the ground-truth. 

IV. Conclusions 

The main contribution of this study is the proposal of a 
novel post-verification framework that achieves good preci¬ 
sion recall trade-off in loop closure detection. We showed the 
loop closure detection can be viewed as an instance of multi¬ 
model hypothesize-and-verify framework, and based on an 
incremental extension of this framework, we built strategies 
for guided sampling, by which loop closures proposed by 
image retrieval are verified in a planned order rather than in 
a conventional uniform order to operate in a constant time. 
Experimental results using a stereo SLAM system confirmed 
that the proposed strategy, the use of loop closure constraints 
and robot trajectory hypotheses as a guide, achieves promis¬ 
ing results despite the fact that there exists a significant 
number of false positive constraints and hypotheses. 
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